Search for: All records

This study explores the potential of the large language model GPT-4 as an automated tool for qualitative data analysis by educational researchers, exploring which techniques are most successful for different types of constructs. Specifically, we assess three different prompt engineering strategies — Zero-shot, Few-shot, and Few-shot with contextual information — as well as the use of embeddings. We do so in the context of qualitatively coding three distinct educational datasets: Algebra I semi-personalized tutoring session transcripts, student observations in a game-based learning environment, and debugging behaviours in an introductory programming course. We evaluated the performance of each approach based on its inter-rater agreement with human coders and explored how different methods vary in effectiveness depending on a construct’s degree of clarity, concreteness, objectivity, granularity, and specificity. Our findings suggest that while GPT-4 can code a broad range of constructs, no single method consistently outperforms the others, and the selection of a particular method should be tailored to the specific properties of the construct and context being analyzed. We also found that GPT-4 has the most difficulty with the same constructs than human coders find more difficult to reach inter-rater reliability on. 

Research on epistemic emotions has often focused on how students transition between affective states (e.g., affect dynamics). More recently, studies have examined the properties of cases where a student remains in the same affective state over time, finding that the duration of a student's affective state is important for multiple learning outcomes. However, the likelihood of remaining in a given affective state has not been widely studied across different methods or systems. Additionally, the role of motivational factors in the persistence or decay of affective states remains underexplored. This study builds on two prior investigations into the exponential decay of epistemic emotions, expanding the analysis of affective chronometry by incorporating two detection methods based on student self-reports and trained observer labels in a game-based learning environment. We also examine the relationship between motivational measures and affective decay. Our findings indicate that boredom exhibits the slowest decay across both detection methods, while confusion is the least persistent. Furthermore, we found that higher situational interest and self-efficacy are associated with greater persistence in engaged concentration, as identified by both detection methods. This work provides novel insights into how motivational factors shape affective chronometry, contributing to a deeper understanding of the temporal dynamics of epistemic emotions. 

Students in open-ended educational games have a number of different pathways that they can select to work productively through a learning activity. Educators and system designers may want to know which of these pathways are most effective for engagement, learning, or other desirable outcomes. In this paper, we investigate which prior jobs and factors are associated with higher rates of student quitting behavior in an educational science exploration game. We use a series of Chi squared analyses to identify the jobs with the highest rates of quitting overall, and we calculate logistic regressions within specific jobs to determine the potential factors that lead to students quitting those jobs. Our analysis revealed that for 23 of the 40 jobs examined, having experience in at least one previous job significantly decreased the chances of students quitting the subsequent job, and that completing specific prior jobs reduces quit rates on specific later jobs. In our discussion, we describe the challenges associated with modeling quitting behavior, and how these analyses could be used to better optimize students’ pathways through the game environment. Specially, guiding students through specific sequences of preliminary jobs before tackling more challenging jobs can improve their engagement and reduce dropout rates, thus optimizing their learning pathways. 

Digital learning games can help address gender disparities in math by promoting better learning experiences and outcomes for girls. However, there is a need for more research to understand why some digital learning games might be especially effective for girls studying mathematics. In this study, we assess two possible pathways: that girls might benefit from math games because they reduce the anxiety and evaluation apprehension that girls are more likely to experience when doing math; and that girls might benefit from math games when they enjoy the narrative and thus experience greater engagement. To evaluate these pathways, our work uses multiple dimensions of gender (e.g., gender identity and gender-typed interests, activities, and traits) and surveys of affective experiences to examine the impact of three learning systems with identical learning content: a digital learning game, Decimal Point, that has consistently led to better learning for girls over boys; a new masculine-typed game, Ocean Adventure, developed based on a survey of over 300 students; and a conventional tutoring system. We predicted that girls and students with stronger feminine-typed characteristics would experience less math anxiety in both Decimal Point and Ocean Adventure compared to the tutor. We also predicted that girls and students with stronger feminine-typed characteristics would experience greater engagement and learning with Decimal Point while boys and students with stronger masculine-typed characteristics would experience greater engagement and learning with Ocean Adventure. Consistent with predictions, students with stronger feminine-typed characteristics experienced less anxiety and evaluation apprehension in both games compared to the tutor. This suggests that math learning games may provide a way to address these negative affective experiences. In terms of our measures of engagement, we found that students with stronger masculine-typed characteristics reported greater experience of mastery in the masculine Ocean Adventure; however, this was the only indicator that the more masculine narrative of Ocean Adventure led to different experiences based on gender. This suggests that narrative alone may not have a strong enough effect on students based on gender, especially when other game features are kept constant. Contrary to our predictions, there were no effects of gender identity or condition on learning outcomes, although both masculine-typed and feminine-typed characteristics were negatively associated with learning. Overall, these results point to the value of a multi-dimensional model of gender in assessing learning with a game, the important role learning games can have in reducing math anxiety and evaluation apprehension for girls and students with feminine-typed characteristics, and the nuanced effects of game narratives on experiences with game-based learning. 

This paper evaluates the use of data logged from cybersecurity exercises in order to predict which students are potentially at risk of performing poorly. Hands-on exercises are essential for learning since they enable students to practice their skills. In cybersecurity, hands-on exercises are often complex and require knowledge of many topics. Therefore, students may miss solutions due to gaps in their knowledge and become frustrated, which impedes their learning. Targeted aid by the instructor helps, but since the instructor’s time is limited, efficient ways to detect struggling students are needed. This paper develops automated tools to predict when a student is having diffculty. We formed a dataset with the actions of 313 students from two countries and two learning environments: KYPO CRP and EDURange. These data are used in machine learning algorithms to predict the success of students in exercises deployed in these environments. After extracting features from the data, we trained and cross-validated eight classifiers for predicting the exercise outcome and evaluated their predictive power. The contribution of this paper is comparing two approaches to feature engineering, modeling, and classification performance on data from two learning environments. Using the features from either learning environment, we were able to detect and distinguish between successful and struggling students. A decision tree classifier achieved the highest balanced accuracy and sensitivity with data from both learning environments. The results show that activity data from cybersecurity exercises are suitable for predicting student success. In a potential application, such models can aid instructors in detecting struggling students and providing targeted help. We publish data and code for building these models so that others can adopt or adapt them. 

The educational data mining community has extensively investigated affect detection in learning platforms, finding associations between affective states and a wide range of learning outcomes. Based on these insights, several studies have used affect detectors to create interventions tailored to respond to when students are bored, confused, or frustrated. However, these detector-based interventions have depended on detecting affect when it occurs and therefore inherently respond to affective states after they have begun. This might not always be soon enough to avoid a negative experience for the student. In this paper, we aim to predict students' affective states in advance. Within our approach, we attempt to determine the maximum prediction window where detector performance remains sufficiently high, documenting the decay in performance when this prediction horizon is increased. Our results indicate that it is possible to predict confusion, frustration, and boredom in advance with performance over chance for prediction horizons of 120, 40, and 50 seconds, respectively. These findings open the door to designing more timely interventions. 

The artificial intelligence in education (AIED) community has produced technologies that are widely used to support learning, teaching, assessment, and administration. This work has successfully enhanced test scores, course grades, skill acquisition, comprehension, engagement, and related outcomes. However, the prevailing approach to adaptive and personalized learning has two main steps. First, the process involves detecting the areas of knowledge and competencies where students are deficient. This process also identifies when or how a student is considered “at risk” or in some way “lacking.” Second, the approach involves providing timely, individualized assistance to address these deficiencies. However, a considerable body of research outside our field has established that such deficit framing, by itself, leads to reactive and less productive strategies. In deficit-based frameworks, powerful student strengths, skills, and schemas—their assets—are not explicitly leveraged...